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[57] ABSTRACT 

A digital image warper system produces a warped output 
image from an input image. The warper system increases the 
sampling rate of a sampled image above the Nyquist rate or 
samples an analog image at a higher rate than the Nyquist 
rate and prefilters the upsampled image to counteract filter- 
ing which may occur during the warping operation. The 
upsampled image is warped using an interpolator for resa- 
mpling to produce a warped upsampled image. The inter- 
polator is, for example, a low quality interpolator such as a 
bilinear interpolator. The warped image is then down- 
sampled to the same resolution as the input image to produce 
the warped image. Down-sampling and warping can be 
combined into one step by modifying the geometric trans- 
formation function implemented when warping the 
upsampled image. 

21 Claims, 10 Drawing Sheets 
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QUALITY IMAGE WARPER 

This non-provisional U.S. national application, filed 
under 35 U.S.C §lll(a) claims, under 35 U.S.C. §119(e) 
(1), the benefit of the filing date of provisional U.S. appli- 
cation Ser. No. 60/015,577, filed under 35 U.S.C. §lll(b) on 
Apr. 18, 1996. 

FIELD OF THE INVENTION 

This invention relates to a method and apparatus of 
digitally processing image data and, particularly, a method 
and apparatus for performing a high quality digital image 
warp. 

BACKGROUND OF THE INVENTION 

There are two basic components to an image warp: (1) 
spatial transformation and (2) resampling through interpo- 
lation. A spatial transformation defines a geometric relation- 
ship between each point in an input and a warped image. 

Inverse mapping is typically used to generate the warped 
image from the input image. The inverse mapping specifies 
reference locations in the source image as a function of the 
current location in the warped image. Commonly, the loca- 
tions in the warped image are processed sequentially to 
generate the warped image from the input image. At each 
integer position in the warped image, the location of a 
corresponding pixel value in the input image is calculated. 
The pixel addresses in the input image may not be integer 
values because the inverse mapping can be arbitrary. In other 
words, the inverse mapping maps a pixel value in the warped 
image to a pixel value in the input image which may be 
located between pixels in the input image. Therefore, it is 
desirable to use interpolation to generate the pixel value 
from the non-integral positions in the input image. 

Typically, several pixels surrounding the referenced input 
location are used to compute the output pixel value in the 
warped image. The larger the number of pixels used, the 
more accurate the resampling. As the number of pixels 
increases, however, so does the cost and complexity of the 
hardware implementation. 

Current hardware approaches for performing the interpo- 
lation required for resampling use a two -by-two neighbor- 
hood of pixels around the address value of the pixel in the 
source image to calculate each pixel value in the warped 
image. This is commonly called bilinear interpolation. Bilin- 
ear interpolation uses a local neighborhood of four pixels 
and a weighted average of those four pixels for the inter- 
polation. For a real-time implementation four pixel values 
are accessed simultaneously every clock cycle. The four 
pixel values are then multiplied by the appropriate weights 
and summed to produce a pixel value in the warped image. 

One such system for performing bilinear interpolation is 
described in Real-time Bilinear Interpolation Using the 
TMC230X by Steve Gomez, TRW LSI Products Division, 
and dated Jan. 21, 1989. This system uses four separate 
memories that can be accessed in parallel when performing 
bilinear interpolation. The system also includes a look-up 
table for storing the coefficients to determine the weighted 
average. The weighting coefficients are multiplied by 
respective pixel values from the input image and, then, 
summed to produce a pixel value in the warped image. 
Real-time Bilinear Interpolation Using the TMC2301 is 
herein incorporated by reference for its teachings on bilinear 
interpolation. 

The difficulty with this and other similar systems is when 
bilinear interpolation is no longer adequate. In applications 
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that require repeated warping of the same image or subpixel 
translations, bilinear interpolation may produce poor results. 
Typically, bilinear interpolators, degrade the high spatial 
frequency components of an image to a greater extent than 

5 a higher quality interpolator. 

The next higher quality interpolator uses a three-by-three 
pixel area in the input image to compute each pixel value in 
the warped image. The complexity and expense of this type 
warper dramatically increases relative to that of a bilinear 

10 interpolator. Nine separate memories, nine coefficients, and 
a nine term sum of products is required. In applications 
where size, power, and cost are at a premium, this is an 
unacceptable solution. If a better interpolation is required, 
for example, four-by-four or greater, this problem is com- 

15 pounded. 

To overcome these shortcomings, a new warper method 
and system is provided. 

SUMMARY OF THE INVENTION 

20 

The present invention is directed to a digital image warper 
system which produces a warped image from an input 
image. The warper system increases the sampling rate of a 
sampled image above the Nyquist rate or samples an analog 
image at a higher rate than the Nyquist rate. The upsampled 

25 image is warped using an interpolator for resampling to 
produce a warped upsampled image. The interpolator is, for 
example, a low quality interpolator such as a bilinear inter- 
polator. The warped image is down sampled to the same 
resolution as the input image to produce the warped image. 

30 Down sampling and warping can be combined into one step 
by modifying the geometric transformation function imple- 
mented when warping the upsampled image. 

It is to be understood that both the foregoing general 

35 description and the following detailed description are 
exemplary, but are not restrictive, of the invention. 

BRIEF DESCRIPTION OF THE DRAWING 

The invention is best understood from the following 
4 0 detailed description when read in connection with the 
accompanying drawings, in which; 

FIG. 1 is a block diagram of an exemplary embodiment of 
the present invention directed to a digital image warping 
system. 

45 FIG. 2 is a block diagram of upsampler 10 shown in FIG. 
1 according to an exemplary embodiment of the present 
invention. 

FIG. 3 is a more detailed block diagram of upsampler 10 
shown in FIG. 2. 
50 FIG. 4 is a data structure diagram illustrating the concep- 
tual operation of upsampler 10 shown in FIGS. 2 and 3. 

FIG. 4a is a representation of the upsampled image signal 
207 where the pixel values in the upsampled image signal 
5s 207 have been renumbered. 

FIGS, 5(a)-5(c) are respective Fourier transforms which 
illustrate exemplary spatial frequency spectra of the input 
image signal 205, intermediate image signal 206, and 
upsampled image signal 207 shown in FIG. 4. 
60 FIG. 6 is a block diagram of warper 20 shown in FIG. 1 
according to an exemplary embodiment of the present 
invention. 

FIG. 7 is a data structure diagram illustrating the input 
image, the upsampled image, and the warped image. 
65 FIG. 8 is a flow chart diagram illustrating the operation of 
warper 20 shown in FIG. 1 in accordance with an exemplary 
embodiment of the present invention. 
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FIG. 9(a) shows respective Fourier transforms of the having a size of 2048 pixel values by 2048 pixel values. The 

upsampled image signal 207, the intermediate warped input image signal 205 is, for example, 256 pixel values by 

image, and the warped image 700 illustrating the operation 256 pixel values. Thus, upsampling by a factor of two 

of the down sampler 30 in accordance with an exemplary produces an upsampled image signal 207 having 512 pixel 

embodiment of the present invention. 5 values by 512 pixel values. 

FIG. 9(b) is a Fourier transform of a input image signal Alternatively, upsampler 10 can be provided an analog 

using a bilinear interpolation which does not include upsam- input signal. In this configuration, upsampler 10 samples the 

pling of the input image signal. analog input signal at a rate of 2 N times the Nyquist rate to 

FIGS. 9(c)-9(0 are enlarged views of the data structure produce the upsampled image signal 207. 
diagram shown in FIG. 7. 10 Upsampler 10 is a high quality interpolation filter shown 
FIG. 10 is a block diagram of another exemplary embodi- in FIG - 2 * Upsampler 10 upsamples the input image signal 
ment of the present invention directed to a digital image 205 b y a factor of two ™ both directions using a separable 
warper implemented on a single integrated circuit (chip). filter - Upsampler 10 slightly peaks the frequency compo- 
rt 11 • j- c\ c i • • nents that will be attenuated by warper 20, shown in FIG. 1, 
FIG. 11 is a timing diagram useful for explaining the i< . . . ... J c * * . 

operation of the warper chip of FIG. 10. to m P rove ,he lml S e of the wal P ed ,ma S e - 

The input image signal 205 is provided to vertical filter 

DETAILED DESCRIPTION OF THE 210 and vertical filter 220 directly and via lines delays 200 

INVENTION eacn °f w hi cn delays an entire line of the input image for one 

line period. Vertical filter 210 is a unity filter and passes the 

Overview 20 pixel values of the even lines of the input image signal 205. 

Vertical filter 220 is a four tap filter with the coefficient 

The exemplary embodiment of the present invention is values se i ecte d to be a sum of the powers of 2. The 

directed to a digital image warper system which produces a coefficient values are -0.09375, 0.59375, 0.59375, and 

warped image from an input image. The warper system -0.09375 which are respectively -3/32, 19/32, 19/32, and 

increases the sampling rate of a sampled image above the 25 _ 3/32 ^ multiplication operations used to implement 

Nyquist rate. Typically, digital images to be warped are vertical fii ter 220 are simplified to arithmetic shift and sum 

sampled at the Nyquist rate. Increasing the sampling rate operations by using these coefficient values, 

reduces the spatial frequencies represented by a fixed num- The ^ values from vertical fi]ters 210 and 220 are 

ber of samples The increased sampling rate is accomplished ^ ^ (1) hodzonta[ filters which are unit fiUers 

by sampling the analog image at a higher rate than the 30 ^ ^ jtelvihu* art (2) boj^tii mttis W 

Nyquist rate or by digitally processing an already Nyquist which arc four tap filters. Tlje coefficient values of horizontal 

sampled image to increase Its sampling rate (e.g by dou- ^ 240 afe to ^ a sum of me eis of 2 and 

bling or tripling each sample). The later of these two arc _ 0 09 3 75> 0.59375, 0.59375, and -0.09375. 

approaches maintains the modularity of the warper system „ . M , , A , , . r , „ _ 

, \ « FIG. 3 is a more detailed block diagram of upsampler 10 

withm a larger image processing system. . n . * K *\ 

shown m FIG. 2. Pixel values from a line of the input image 

The upsampled image is warped, using an interpolator for signa] 205 are provided to inmit terminal IN> ^ Hnes of thc 

resampling, to produce a warped upsampled image. Hie k t { si d 205 are processed throu ^ h Hne delays 

interpolator can be a relatively low quality interpolator such 200> adders 30 o«-300/, and multipliers 310«-310e to ver- 

as a bilinear interpolator The warped image is then down ^ mt late me odd lines of tne i t ^ and 

sampled to the same resolution as the input image to produce duce vcrtically interpolated lines odd which become the 

the warped image. Down sampling and warping can be odd numbered lines of the upsampled image 207, shown in 

combined mto one component by modifying the geometric FIG x Coefficients C0) a and a are y 16 , and 

transformation function implemented when warping the respectively. The output signal Veven of the upsampler 10 

upsampled image. ^ are nQt vertically filtered. These lines become the even 

DESCRIPTION OF EXEMPLARY numbered lines of the upsampled image 207 and represent 

EMBODIMENTS image samples or the mput image as they were received. 

The even lines Veven are provided to and processed using 
Referring now to the drawings, wherein like reference pixel delays 320a-320</ } adders 300&-300/, and multipliers 
numerals refer to like elements throughout, FIG. 1 illustrates 50 310/-310/ to horizontally interpolate the pixels in the even 
the digital image warping system 5 according to an exem- lines Veven to produce output pixel signal OUT2 which 
plary embodiment of the present invention. An input image represents the odd pixels in the even lines of the upsampled 
signal 205 is provided to upsampler 10. The input image image 207, shown in FIG. 1. The even pixels in the output 
signal 205 is sampled at or above the Nyquist rate and is a image signal are not horizontally interpolated and are pro- 
complete representation of the underlying analog image. 55 vided as output pixel signal OUT1. The vertically interpo- 
Upsampler 10 interpolates the input image signal 205 by a lated lines Vodd are provided to and processed using pixel 
factor of, for example, 2 N , where N is a positive integer delays 320e-320g, adders 300m-300r, and multipliers 
greater than zero, to increase the number of samples, i.e. 310;-310« to produce horizontally interpolated odd pixels in 
pixel values, in the input image signal 205. Interpolation is the vertically interpolated lines Vodd to produce output pixel 
not limited to a factor of 2^. 60 signal OUT4. The pixels of the vertically interpolated lines 
Upsampling is performed in the vertical and horizontal Vodd are also provided, without further interpolation, as 
directions of the input image signal 205 using the same pixel signal OUT3. 

sample multiplication factor. For example, if the input image FIG. 4 illustrates the conceptual operation of upsampler 

205 has 512 pixel values by 512 pixel values, upsampling by 10 shown in FIGS. 2 and 3. FIG. 4 is a spatial domain 

a factor of two produces an upsampled image signal 207 65 representation of the sampled images which illustrates 

having 1024 pixel values by 1024 pixel values. Upsampling upsampling by a factor of two (N =2). Zero data values are 

by a factor of four produces an upsampled image signal 207 inserted into the input image signal 205, as shown in FIG. 4, 
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to produce an intermediate image signal 206. A, B, C, and bilinear interpolation. Because the warp memory 600 is 

D represent pixel values in the input image signal 205. implemented as four separate memories 600a-600t/, for use 

Intermediate image signal 206 is low-pass filtered to pro- by the bilinear interpolation, the upsampled image signal 

duce upsampled image signal 207 having pixel values A, B, 2 07, shown in FIG. 4, can be stored in the warp memory 600 

C, and D plus interpolation pixel values L By implementing 5 at four pixels dock < , n ^ nQ badd d< . 

this conceptual process, the Nyquist sampled image 205 is fe incUfred . Q ^ ler 10 shown [n FIG L 
cntically sampled. 

Inserting zeros into the input image signal 205 com- Bilinear interpolation uses a local neighborhood of four 

presses the Fourier transform of the image by a factor of two pixels and a weighted average of those four pixels for the 

as is shown in FIGS. 5(a)-5(c). FIGS. 5(a)-5(c) are respec- J° interpolation. The neighborhood of pixel values are the 

tive Fourier transforms of the input image signal 205, nearest pixel values to the source pixel value. The source 

intermediate image signal 206, and upsampled image signal pixel value 720 and the four closest neighbor pixel values 

207. A comparison of FIGS. 5(a) and 5(c) demonstrates that 72 Q a -720d in the upsampled image are shown in FIG. 7. 

region 400 shown in FIG, 5(«), over which the Fourier j t image 205 ^ upsampled image 2 07 have a pixel 

transform of the input image signal 205 extends is reduced 15 ^ at ^ ^ horizQntal and 

in size to region 401 in the upsampled image signal 207, showfl ? pkel ^ 

shown m FIG. 5(c). Further, as is shown in FIG. 5(c), a , . „_ ... j r . . 

. t c f < j . P . * j warped image 200 which corresponds to source pixel 720. 

single copy of the compressed transform centered around _. \ ~ , 

every integer multiple of 2* remains after the intermediate Pixel 700 also ""responds to pixel 710 in the input image, 

image 206 is low pass filtered. 20 in a real-time implementation, four pixel values (720a, 

The upsampling kernel is a low-pass filter, with a cutoff 720fc, 720c and 120d) stored in the warp memory are 

frequency determined by the extent to which the input image retrieved at substantially the same time every clock cycle for 

signal 205 is upsampled. The low-pass filter eliminates use by the bilinear interpolator. These pixel values are then 

frequencies except those compressed copies of the original multiplied by the appropriate weights and summed to pro- 

signal that are centered on integer multiples of 2ji. If the 25 duce a pixel value in the intermediate warper image. The 

frequency characteristics of the interpolation method pixel values of the upsampled image signal 207 are stored in 

employed in the warper 20, shown in FIG. 1, are known, a mem ories 600a-600c/ so that every grouping of the nearest 

better upsampling kernel can be selected to provide an pixe] values 720 a-720d to the source pixel value may be 

overall warper system 5 with less image degradation. ^ accessed concurrent ly. 

The upsampling kernel is selected to compensate for the 

deviations from a perfect low-pass reconstruction filter that Write address generator 615, shown in FIG. 6, produces 

may exist in the warper 20. For example, if the warping addresses for storing the pixel values in the upsampled 

interpolation method is bilinear, described below, after image signal 207 in memories 600a-600<*. Upsampler 10, 

upsampling, there will be a slight attenuation of the high 35 shown in FIG. 1, produces a two -by-two array of pixel 

spatial frequencies. The upsampling kernel of upsampler 10 values [(A A/ , E kl , F kl , G k /( ), (B^ ; , E kl , F^, G^,,), (C^, E kJ , 

is chosen to slightly peak these same frequencies to mini- F w , G k l7 ), or (D M , E kJ , F*,, G w ,)] for each pixel value in the 

mize the overall warper system 5 effect on the high spatial input image signal 205, shown in FIG. 7, where k and 1 are 

frequencies. Thus, if the warp interpolation method is integers. The four pixels in the two-by-two array are stored 

known, an upsampling kernel for upsampler 10 can be 4Q in separate memories 600a-600^ at the same address in each 

selected to compensate for these deviations from a perfect memory 600a-600<* and at substantially the same time. In 

low-pass filter by pre-amplifying or pre-attenuating frequen- the exemplary embodiment of the invention this is imple- 

cies in the input image signal 205. In this way, the overall mented b £iod each of (he QUt t ^ QUT1> 

degradation of the system can be reduced. Qim> QTjn and QTJT4 of ^ upsampler 10 shown m 

Different warp transforms attenuate different high spatial 45 FIG 3 ^ a resp ective memory 600a, 600b, 600c, and 600& 
frequencies when bilinear interpolation is used. For 

example, a half-pixel translation causes the most attenuation For example, assume the input image has 256 pixel values 

of the high spatial frequencies while integer translations by 256 pixel values and the input is upsampled by a factor 

have no attenuation. A single upsampling kernel between of two. Then the upsampled image 207, shown in FIG. 7, has 

these to extremes can be selected to improve the overall 50 5 12 pixel values by 512 pixel values where each square 

image quality for different transforms. grouping of pixels 210, shown in FIG. 4, has a pixel located 

The upsampler generates pixel values in the upsampled in a separate memory. The pixels values shown in FIG. 4 are 

image signal 207 at four times the rate pixels in the input stored in memories 600a-600(i. FIG. 4a is a representation 

image are provided to upsampler 10. Pixel values from the of the upsampled image signal 207 where the pixel values in 

input image signal 205 are provided to upsampler 10, shown 55 the upsampled image signal 207, shown in FIG. 4, have been 

in FIG. 1, at a rate of one pixel per clock cycle and supplied renumbered as Zl through Z4 for describing where the pixel 

from upsampler 10 at four pixels per clock cycle. The four values in the upsampled image are stored in memories 

pixels are provided to warper 20, shown in FIG. 1, and 600a-600d. 
stored in, for example, separate memories 600a-600d, 

shown in FIG. 6, as explained below. 60 Each memory address Xp and y p is twelve bits if the 

Warper 20, shown in FIG. 1, warps the upsampled image maximum size is 2 12 x2 2 (4096x4096). Each memory 

signal 207, shown in FIG. 4. Warper 20 is described with address x p and y p includes a least significant address bit x 0 

reference to FIG. 6. In FIG. 6, the pixel values of the and y 0 which together specify the memory 600a-600c/ in 

upsampled image signal 207 are stored in memories which the pixel value is stored and location address bits x u .j 

600a-600rf so than each one of the memories stores one 65 and y 11 . 1 which together specify the address at which the 

fourth of the upsampled image 207. The pixels in the warped pixel values are stored. The mapping of address bits x 0 and 

image are interpolated from the stored pixel values using y 0 into memories 600a-600^ is shown in Table 1 below. 
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TABLE 1 



8 





y n 


MEMORY 


0 


0 


600a 


0 


i 


600b 


1 


0 


600c 


1 


i 


600d 



FIG. 4a and Table 2 below illustrate the arrangement of 10 
the pixel values of the upsampled image 207 in memories 
600a-600</ addresses x[ll -1] and y[ll-l]. 



Equations (1) and (2) are implemented using the pseudo 
code in Table 3 below. 

TABLE 3 

/"[nitialize at the start of image warp:*/ 
x[0,0] = a x 
accejO] *» 0 
accfjO] - 0 
accejl]- fr* + 

/* Compute before the start of each line in the 
warped image:*/ 
accejvj - accejv-l] + 2e* 



TABLE 2 



x[ll-l] 


y[ll-l] 


MEMORY 
600A 


MEMORY 
600B 


MEMORY 
600C 


MEMORY 
600D 


0 


0 


Zl 00 


Z2o, 0 


Z3o,o 


Z4 0 ,o 


0 


1 






Z3 0 , a 


Z4 0il 


0 


2 


Zlo! 2 


Z2o. 2 


Z3 0>2 


Z4 0 . 2 


0 


255 


Zlo,255 


Z2o,255 


Z3o,253 


2*0,255 


1 


0 






Z3 1>0 




1 


1 


Zli,i 


Z2 14 


Z3 W 


Z4 M 


1 


255 


Z3i^55 


Z2 lt 255 


Z3 1,2)5 


Z4 li255 


2 


0 


Zl 2 , 0 


Z22,o 


Z32.0 


Z4 2 . 0 


255 


255 


Zl 255 ,255 


Z2255.255 


Z3255.255 


Z4-255.255 



For example, memory address [(x^^, x 0 ), (Y^.j, Y 0 )] a 
[(1, 0), (255, 0)] is pixel value Zlj 255 stored in memory 
600a. 

Once the upsampled image signal 207 has been stored in 
memories 600a-600d, it may be warped by warper 20. The 
generation of pixel value 700 in the warped image 208, 
shown in FIG. 7, is described below with reference to FIG. 
8. 

First, at step 800, the address value A[x,y] of the source 
pixel value 720 in the upsampled image is generated, x and 
y are coordinates of the source pixel value 720 in the 
upsampled image and are used as an address to memories 
600a-600d described above with reference to Tables 1 and 
2. In response to the transform to be implemented, address 
generator 620 generates an address value A[x f y] for the 
source pixel value 720 in the upsampled image 207, shown 
in FIG. 7. For example, the input image 205 can be trans- 
formed in the spatial domain using a geometric transforma- 
tion illustrated in equations (1) and (2) below. 

-4 w, v^a x +b x - u+c x -v+d x -u z +e x - v*+f/wv (1) 
y[u, v\oa y +b y - u+c y -v+d y u 2 +e y - A/yu'V (2) 

u and y are coordinates of a pixel value in the intermediate 
warped image signal and a,,, b^, c^, d x , e^, f^ a y , b y , c y , d y> ey> 
and fy are constant coefficient values. The transform equa- 
tions are provided to the address generator 620 via control 
signal CS1 which is provided via an interface (not shown). 
Alternatively, the transform equations can be stored in a 
memory (not shown) in address generator 620. 



35 

TABLE 3-continued 

x[0,v] - x[0,v-l] + accejv] 
accfj v] - accfjv-l] + f x 
4Q /initialize before each line of the warped image 

and after step 20: */ 
accdjO] - 0 

accdjl] = b x + dx + accfjv] 

/"Compute for each pixel in the warped image:*/ 

accdju] = accdju-l] + 2d^ 

x[u,v] «* x[u-l,v] + accdju] 



In Table 3, x[u,v] is the x-coordinate of address value A[x,y] 
in the upsampled image 207 which corresponds to a pixel 
value in the intermediate warped image signal shown in FIG. 

50 1. This intermediate signal corresponds to the warped image 
208 prior to down sampling by down sampler 30 shown in 
FIG. 1. The y-coordinate y[u,v] is generated using the 
pseudo code in table 3 except x[u,v] in Table 3 is replaced 
with y[u,v] and the "x" subscripts are replaced with "y" 

55 subscripts. 

The pseudo code in Table 3 can be implemented in 
hardware using two adders including four accumulator reg- 
isters for each computation to produce values x[u,v] and 
y[u,v]. The precision of the address generation is high, for 

60 example, 32 to 48 bits, depending on the system require- 
ments and accuracy required for the generation of the 
warped image 208, shown in FIG. 7. 

The address value A[x,y] has an integer part and S y , 
and a fractional part T* and T y because the address value 

65 A[x,y] identifies the location of the source pixel 720 in the 
upsampled image 207, shown in FIG. 7, with sub-pixel 
precision. S x and S y are the respective integer parts of the 
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coordinate values x and y and T x and T y are the respective 
fractional parts of the coordinate values x and y. The integer 
part S x and of the address value A[x,y] is used to select 
the two-by-two pixel neighborhood of the source pixel value 
at address value A[x,y]. The fractional part T* and T y of the 5 
address value A[x,y] is used to compute the interpolation 
coefficients that are provided to the bilinear interpolator. 
Interpolation coefficient values weight the pixel values in the 
two-by-two pixel neighborhood to produce the pixel value in 
the intermediate warped image shown in FIG. 1. 10 

At step 810, after the address value A[x,y] has been 
generated, it is determined whether the source pixel 720 
corresponds to one of the pixel values in the upsampled 
image signal 207. If so, the pixel value in the warped image 
is selected as the pixel value in the intermediate warped 35 
image signal. For example, if the address value A[x,y] 
identifies pixel value 720a, shown in FIG. 7, as the source 
pixel, then pixel value 720a is selected. Otherwise step 830 
is performed. 

At step 830, the two-by-two pixel neighborhood 20 
120a-720d of the source pixel 720 is selected. The above - 
referenced article by Gomez describes one method for 
selecting and retrieving the two-by-two pixel neighborhood 
from four separate memories. 

As shown in FIG. 7, source pixel 720 is located within a 25 
two-by-two neighborhood of pixels 720a-720rf which are 
used to produce the pixel value in the intermediate warped 
image signal. The four pixels 120a-720d which make up the 
two-by-two neighborhood are located in separate memories 
600a-600d. FIG. 9 is an enlarged view of a portion of the 30 
upsampled image 207 shown in FIG. 7. Address generator 
620 uses the upper left-most pixel as a nearest neighbor of 
the two-by-two neighborhood. The nearest neighbor is the 
starting position from which the address value for each pixel 
value 120a-720d in memories 600«-600d* is determined. 
FIGS. 9(c)-9(/) illustrate the four different positions of 
source pixel 720 as it relates to pixel values 720a-720d 
which form the two-by-two neighborhood. A pixel value of 
the upsampled image is located at each vertex created by the 
vertical and horizontal lines in FIGS. 9(c)-9(/). FIGS. 9(a) 





10 






TABLE 5 




Memory 


X Component Address 


Y Ccmponent Address 


Memory 600a 
Memory 600b 
Memory 600c 
Memory 600d 


SJIM] + 1 
SJIM] 
SJIM] + 1 
SJIM] 


SJIM] 
S y [lM] 
SyllM] 
S y [lM] 



The addresses of pixel values 720a-720rf in memories 
600a-600d* are determined in accordance with Table 6 if the 
nearest neighbor to source pixel value 720 is stored in 
memory 690c. 



TABLE 6 



Memory 


X Component Address 


Y Component Address 


Memory 600a 


SJIM] 


S y [ll-1]+ 1 


Memory 600b 


SJIM] 


S y [lM]+ 1 


Memory 600c 


SJIM] 


S y [lM] 


Memory 600d 


SJIM] 


S,[1M] 



The addresses of pixel values 120a-120d in memories 
600a-600a' are determined in accordance with Table 7 if the 
nearest neighbor to source pixel value 720 is stored in 
memory 600d, 



TABLE 7 



Memory 


X Component Address 


Y Component Address 


Memory 600a 
Memory 600b 
Memory 600c 
Memory 600d 


SJIM] + 1 
SJIM] 
SJIM] + 1 
SJIM] 


S y [lM] + 1 
S y [ll-1] + 1 
SyllM] 
S y [lM] 



35 



After the addresses of each of the pixel values 720a-720d 
stored in memories 600a-600rf are determined, the pixel 
values 720a-720rf are retrieved from memories 720a-720o", 



Next, at step 840, the interpolation coefficient values II, 
■JK^N^si^ Vach pixel 40 K > ^ d 14 are Produced using the fractional parts T and 

— T y of address value A[x,y], Interpolation coefficients II, and 

12, 13, and 14 are produced using equations (3)-(6) below. 



value 720a-120d of the two-by-two neighborhood is stored. 

The integer part and S y of the address value A[x,y] of 
the source pixel value 720 is used to determine the addresses 
of the two-by-two neighborhood in memories 600a-6O0d. 
Particularly, the least significant bit of the integer part SJ0] 
and S^[0] are used to determine the addresses. When, as 
shown in FIG. 9, the nearest neighbor is stored in memory 
600a, the address of each one of the pixel values 720a-720d 
in the two-by-two neighborhood is determined in accor- 
dance with Table 4 below. 

TABLE 4 



45 



n-(i-T,>(i-T y ) 

I2=T,-(1-T,) 
I3=(1-TJ-T > , 
I4-T r -T„ 



(3) 
(4) 
(5) 
(6) 



Memory 


X Component Address 


Y Component Address 


Memory 600a 


SJIM] 


SJIM] 


Memory 600b 


SJIM] 


SJIM] 


Memory 600c 


SJIM] 


SJIM] 


Memory 600d 


SJ3M] 


SJIM] 



In Table 4, SJll-1] are the eleven most significant bits of 
the x component part calculated using equation (1) and 
S^fll-l] are the eleven most significant bits of the y com- 
ponent part calculated in accordance with equation (2). 

The addresses of pixel values 720a-720d in memories 
600a~600d are determined in accordance with Table 5 if the 
nearest neighbor to source pixel value 720 is stored in 
memory 600fc. 



55 



60 



Alternatively, interpolation coefficient values 11-14 can be 
stored in a look-up table in coefficient generator 630. The 
fractional parts T x and T y of the address value A[x,y] are 
used as an address to the interpolation coefficient values 
stored in the look-up table. The above-referenced paper by 
Gomez describes one method for using the fractional parts 
T x and T v of the address value A[x,y] as an address for 
retrieving interpolation coefficient values from a look-up 
table. 

Returning to FIG. 8, at step 850, the source pixel value 
720 is produced using the interpolation coefficients 11-14 
and the two-by-two neighborhood of pixels 720a-720a\ 
shown in FIG. 7, retrieved from memories 600a-600o\ As is 
shown in FIG. 6, pixel value 720a from memory 600a and 
interpolation coefficient value II are multiplied at multiplier 
610a. Pixel value 720& from memory 600b and interpolation 
coefficient value 12 are multiplied at multiplier 6106. Pixel 
value 720c from memory 600c and interpolation coefficient 
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value 13 are multiplied at multiplier 610c. Pixel value 720d In one implementation, the timing signals HA and VA 

from memory 600d and interpolation coefficient value 14 are remain associated with the input image signal as the input 

multiplied at multiplier 610rf. The output values provided image signal is processed by the warper chip 1000. As a 

from multipliers 610a-610d are added together by sum result, when the warped image signal is produced, the 

circuitry 640 to produce source pixel 720, shown in FIG. 7. 5 vertical and horizontal blanking periods of the warped image 

At step 860 in FIG. 8, the source pixel 720 is selected as si S nal do not need 10 be generated because the relative 

a pixel value in the intermediate warped image signal. Steps timm g of the war P ed image signal is maintained by associ- 

800 through 860 are then repeated to produce each pixel in a ! ia S the timing signals HA and VA with the input image 

the warned irnaee 208 signal as it is processed. Thus, less circuitry is required. In 

~ ^ , .* . c , 4 , another implementation, the upsampling and the storing 

Down sampler 30 reduces the number or samples in the 10 r . i j «. .u • j * . i 

. . A v . . * * r c 41 process may be uncoupled from the warping and interpola- 

intermediate warped image signal by the same factor the tion process, in which case the HA and VA timing signals for 

input image signal is upsampled. For example, if the input ^ t have tQ be re . gen erated. A further description of 

image signal is upsampled by two, then the intermediate the use of me timiog signals fc omittedt 

warped image is down sampled by two. FIG. 9(a) shows ^ ex6mp lary embodiment is not limited to processing 

respective Fourier transforms of the upsampled image signal is mu lti-bit digital signals containing eight data bits and two 

207, the intermediate warped image, and the warped image timing bits. The exemplary embodiment can be configured 

700 illustrating the operation of the down sampler 30. FIG. to receive multi-bit digital signals containing, for example, 

9(b) shows Fourier transform of a input image signal using ten data bits and two timing bits. 

a bilinear interpolation which does not include upsampling Referring to FIG. 10, the input image signal is provided 

the input image signal. A comparison of the warped image 20 to delay 1010 and upsampler 1015. Delay 1010 is the same 

signals shown in FIGS. 9(a) and 9(b) illustrates that the as delay 200 shown in FIG. 2 and upsampler 1015 is the 

warped image signal produced using upsampling prior to combination of vertical filter 210 and 220, shown in FIG. 2, 

transformation and down sampling after transformation pro- and horizontal filters 230 and 240, shown in FIG. 2. Delay 

duces a warped signal with better frequency characteristics 1010 and upsampler 1015 operate the same as these corn- 

than a warped signal which does not use upsampling and 25 ponents. Memories 600a-600</ are the same as memories 

down sampling. 600a-600rf shown in FIG. 6 and are a single or double bank 

Alternatively, the down sampler 30 and the warper 20 can of SRAM. Bilinear interpolator is a combination of multi- 
be combined into a single component or step. In this case, pliers 610«-610rf and sum circuitry 640 shown in FIG. 6. 
the address value A[x,y] is adjusted by the interpolation Address/coefficient generator 1050 operates the same as 
factor 2^ to account for the interpolation of the input image 30 address generator 630 and coefficient generator 630 shown 
signal 205. For example, the standard geometric transfor- in FIG. 6 except that address/coefficient generator 1050 may 
mation specifies that a pixel value in the warped image also receive a flow field input. The flow field input provides 
signal should come from a pixel value in the input image a two-dimensional vector for each output pixel. The flow 
signal at location A[x=Xl, y=Yl). This pixel value is at a field vector is added to the warp address generated by the 
non-integral location in the input image and, thus, is deter- 35 address/coefficient generator 1050 to allow completely gen- 
mined by interpolation using the neighborhood pixels. The eral image transformations or to provide general modifica- 
same pixel value in the upsampled image signal 207 is tions to the computed data of the address/coefficient and 
accessed by multiplying the referenced address by 2. In this generator. Address/coefficient generator 1050 also receives 
case, the address value to reference in the upsampled source control signals CS1 from CPU interface 1040. Warp coef- 
image would be A[x«2-xl, y«2Yl). If the image was 40 ficients such as constant coefficient values a^ b^, C^, d^., e x , 
upsampled by a factor of 4, the referenced address would be f x , a^, b^, C y , d v , e^, and fy are provided to generate address 
multiplied by 4. Then a neighborhood of pixels around this values A[x,y]. Address values A[x,y] are used to retrieve 
address is used to determine the output value. The neigh- pixel values from memories 600a-600c/ as described above 
borhood of pixel values in the upsampled image corresponds with reference to write address generator 615 shown in FIG. 
to sample points of the original analog image that are closer 45 6. Write address generator 1020 is also coupled to CPU 
to the location of the source pixel than an equal sized interface 1040 which provides data that specifies which 
neighborhood of pixel values in the input image signal. segment of the input image is to be processed and stored in 

FIG. 10 is another exemplary integrated circuit embodi- memories 600a-600d. Write control logic 1020 provides the 

ment of the present invention directed to a warper chip 1000. write address shown in Tables 1 and 2 for writing the pixel 

Warper chip 1000 is provided with the input image signal 50 values in the upsampled image to memories 600^-600^. 

which is a multi-bit digital signal containing at least eight Based on the timing signals HA in and VA^ and the program 

data bits and two timing bits. The two timing bits convey interface which specifies which segment of the input image 

respective timing signals. One signal, HA^, is in a logic high signal should be stored, the write control logic 1020 gener- 

state when the data in a line is valid (i.e. during the active ates the addresses to store the pixel values in memories 

line interval) and in a logic low state otherwise (i.e. during 55 600a-600rf in the same manner as described above, 

the horizontal blanking interval). The other signal, VA,-„, is In another exemplary embodiment of the present 

in a logic high state when the data in a field is valid and in invention, the circuitry shown in FIGS. 1 and 10 may be 

a logic low state otherwise (i.e. during the vertical blanking used to perform multiple image plane warping. In many 

interval). FIG. 11 is a timing diagram which illustrates the applications it is desirable to warp multiple image planes 

relative timing of the input image signal and the warped 60 such as the separate planes of a color image. This can be 

image signal, and the signals HA and VA for those signals. achieved by using multiple warper chips 1000, shown in 

The warper chip uses the timing signals HA and VA in the FIG. 10, or multiple warper systems 5, shown in FIG. 1, in 

same manner as described in U.S. Pat. No. 5,359,674, parallel. Alternatively, if the processing rate of the warper 

entitled PYRAMID PROCESSOR INTEGRATED chip 1000 or warping system 5 is fast enough, the warping 

CIRCUIT, issued to van der Wal, which is herein incorpo- 65 operations on part or all the image planes can be interleaved, 

rated by reference for its teachings on processing signals As a result, a single warper chip 1000 or warper system 5 

having timing bits. can be used. 
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Interleaving multi-plane images is described below using 
a RGB input image signal and FIG. 10. The input image 
signal includes, for example, three image planes R, G, and 
B, where the image planes R, G, and B are interleaved for 
every pixel for each line of the input image signal as shown 
in Table 8 below 

TABLE 8 

LineO R(0,0] G{0,0] B[0,0] R[1,0] Gtl,0] B[1,0] 



10 



linen R[0,n] GjO,n] B[0,n] R[l,n] G[l,n] B[l,n] 

By increasing line delays 1010 by a factor of three or 
using three line delays for each line delay 200, upsampler 
1015 can operate on the input image signal in an interleaved 
fashion. The upsampled image is stored in memories 
600a-600c/ in the same manner as described above except 
that the pixel values are stored in an interleaved fashion. 
Table 9 illustrates how the pixel values of the three planes 
are stored in memories 600e*-600^. 

TABLE 9 



MEMORY 


MEMORY 


MEMORY 


MEMORY 


600A 


600B 


600C 


600D 


^0,0 




^0.0(0) 


^0(0),0(0) 


G 0.0 


G 0<0).0 


^0,0(0) 


G 0(0).0(0) 


B 0.0 


B o(o).o 


Bo.o(o) 


B 0f0).0(0) 



Rm^j 




Rm,n(0) 






G m (o),* 




Gju(0>(0) 




iW- 


B m ,ti(0) 


B„( 0 ),«(0) 




Rm(D)+l,n 


^mvl,a(0) 


Rn(0)f l,n(0) 


G m+ l.o 






G m (o>*.l,ii(0) 


Bm+t,n 


Bin(o)+i,o 


Bm+l,n(0) 


Bm(0>l,n<D) 



^255.255 
^255 ,255 
^255,255 



1*255(0), 255(0) 
^255(0), 255(0) 



D 255(0), 255 



° 255,255(0) 
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In Table 9 "(°)" represents the odd horizontal and vertical 
outputs of upsampler 1015. The integer part of the address 45 
value A[x,y] generated by address/coefficient generator 
1050 is multiplied by three to account for the change in 
address created by the addition of the other image planes in 
memories. In addition, address/coefficient generator 1050 
repeats the same operation three times: once for the calcu- 50 
lated address value A[x,y], then for the address value A[x,y] 
incremented by one, and then for the address value A[x,y] 
incremented by two. The warped image signal includes color 
image data in an interleaved fashion. 

For the above example, the warper chip operates at three 55 
times the clock rate of the incoming pixel rate of a single 
image plane. If the RGB image planes in the input image 
signal are only available in parallel data paths, the image 
data is interleaved prior to processing. After the warper 
image signal is produced, the warped image signal is 60 
de-interleaved to restore the image planes to parallel data 
paths. 

Another example of interleaved operation of the warper 
chip 1000 is described below where the input image signal 
is an color image in Dl format, where the three image data 65 
planes are interleaved in a slightly different manner. The 
input image signal includes, for example, three image planes 



Y, U, and V, where the image planes Y, U, and V are 
interleaved as . . . U[n,m], Y[n,m], V[n,m], Y[n+l,m], 
U[n+2,m], Y[n+2,m], V[n+2,m], Y[n+3,m], . . . Thus the 
luminance plane Y has twice the resolution of the two color 
planes U and V, and the color planes U and V are co-located 
with the even positions of the luminance plane Y. 

This interleaving can be maintained throughout the 
warper implementation, but the delays and address calcula- 
tions are slightly different then when operating on the three 
image planes as described above. The line delays 1010 prior 
to the upsampler 1015 are twice the size of a line of the 
luminance plane Y which is four times the size of one line 
of the color planes U and V. The delay switches every clock 
cycle between a two-delay element and a four-delay ele- 
ment. 

The address/coefficient generator, in turn computes a new 
address value A[x,y] twice for four successive output values: 
first for the two color planes U and V and the even position 
of the luminance plane Y, and then an address value A[x,y] 
is separately calculated for the odd position of the luminance 
plane Y The address value A[x,y] is also offset to appro- 
priately access the correct data planes stored in memories 
600a-600<* 

Although illustrated and described herein with reference 
to certain specific embodiments, the present invention is 
nevertheless not intended to be limited to the details shown. 
Rather, various modifications may be made in the details 
within the scope and range of equivalents of the claims and 
without departing from the spirit of the invention. 

The invention claimed is: 

1. A image processing system comprising: 

means for applying an input image signal to the image 
processing system, said input image signal having an 
input sampling rate; 

means for uniformly upsampling and filtering all pixels in 
the input image to increase the sampling rate of the 
input image signal to at least two times the Nyquist 
rate, the upsampled image having interpolated values 
spaced uniformly at predetermined distances from each 
input sample; 

means for warping the uniformly upsampled image signal 
to produce an upsampled warped image signal, and 

means for down sampling the upsampled warped image 
signal to produce a warped image output signal having 
a sampling rate which is substantially the same as the 
input image sampling rate. 

2. The image processing system according to claim 1, 
wherein the means for warping includes a bilinear interpo- 
lator. 

3. The image processing system according to claim 1, 
wherein the upsampling and filtering means samples the 
input image signal at a rate of 2 N times the Nyquist rate 
where N is a positive integer. 

4. The image processing system according to claim 1, 
wherein the input image signal has pixel values which are 
provided to the upsampling and filtering means at one pixel 
per clock cycle and the upsampling and filtering means 
produces four upsampled pixel values per clock cycle. 

5. The image processing system according to claim 4, 
wherein the warping means includes four memories and the 
four upsampled pixel values are stored in respectively 
different ones of the four memories. 

6. The image processing system according to claim 1, 
wherein the upsampling and filtering means includes a filter 
for processing the input image signal to counteract deterio- 
ration in spatial frequency characteristics caused by the 
warping means. 
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7. The image processing system according to claim 1, 
wherein: 

the input image signal includes pixel values; 
the warping means includes four memories; and 
the upsampling and filtering means further produces four 
pixel values for each of one the pixel values in the input 
image signal and stores each one of the four pixel 
values in a respectively different one of the four memo- 
ries. 

8. The image processing system according to claim 1, 
wherein: 

the input image signal includes pixel values; and 

the upsampling and filtering means further produces four 

pixel values for each one of the pixel values in the input 

image signal where: 

(a) a first one of the four pixel values corresponds to 
one pixel value in the input image signal, 

(b) a second one of the four pixel values is produced by 
horizontally interpolating ones of the pixel values in 
the input image signal, 

(c) a third one of the four pixel values is produced by 
vertically interpolating ones of the pixel values in the 
input image signal, and 

(d) a fourth one of the four pixel values is produced by 
horizontally and vertically interpolating ones of the 
pixel values in the input image signal. 

9. The image processing system according to claim 1, 
wherein the input image signal includes multiple image 
planes and the image processing means includes means for 
spatially interleaving the multiple image planes and wherein 
the warping means concurrently warps the multiple image 
planes in an interleaved operation. 

10. A image processing system according to claim 1, 
wherein: 

the means for warping and the means for down sampling 
operate at substantially the same time. 

11. An integrated circuit comprising: 

an input terminal to which an input image signal including 
pixel values are provided said input image signal 
having an input sampling rate; 

upsampling and filtering circuitry, coupled to the input 
terminal, which upsamples the input image signal to at 
least two times the Nyquist rate to provide an increased 
sampling rate image signal including pixel values, the 
increased sampling rate image signal having interpo- 
lated pixel values spaced uniformly over the entire 
input image signal at predetermined distances from 
each input pixel value; 

write address circuitry which generates a write address 
signal that specifies a location in a memory where the 
pixels values are to be stored; 

read address circuitry which generates a read address 
signal that specifies which ones of the pixels values are 
to be retrieved from the memory; 

warper circuitry which receives retrieved pixel values 
from the memory and produces a warped image signal; 
and 

down sampling circuitry, coupled to the warper circuitry, 
which produces a warped image output signal, said 
output signal having an output sampling rate substan- 
tially equal to said input sampling rate. 

12. The integrated circuit according to claim 11, wherein 
the down sampling circuitry and the warper circuitry are 
combined as one circuit. 

13. A method for processing an input image signal having 
an input sampling rate comprising the steps of: 
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(a) increasing the sampling rate of the input image signal 
to at least two times the Nyquist rate by interpolating 
values spaced uniformly over the entire input image 
signal at predetermined distances from each input 
sample; 

(b) warping the upsampled image signal, and 

(c) down sampling the upsampled warped image signal to 
produce a warped image output signal having an output 
sampling rate substantially equal to said input sampling 
rate. 

14. The method according to claim 13, wherein step (a) 
samples the input image signal at a rate of 2^ times the 
Nyquist rate where N is a positive integer. 

15. The method according to claim 13, wherein step (a) 
includes the step of filtering the input image signal to 
counteract deterioration in the frequency characteristics of 
the upsampled warped image signal caused by step (b), 

16. The method according to claim 13, wherein: 
the input image signal includes pixel values; 

step (a) produces four pixel values for each one of the 
pixel values in the input image signal, and 

the method further comprises the step of storing each one 
of the four pixel values in a separate memory. 

17. The method according to claim 13, wherein steps (b) 
and (c) are performed at substantially the same time. 

18. An image processing system comprising: 

means for applying an input image signal to the image 
processing system, said input image signal having an 
input sampling rate; 

means for uniformly upsampling and filtering to increase 
the sampling rate of the input image signal, the 
upsampled image having interpolated values spaced 
uniformly at predetermined distances from each input 
sample; 

means for warping the upsampled image signal, and 
means for down sampling the warped image signal to 
produce a warped image output signal; wherein: 

(a) the input image signal is a digital signal sampled at 
the Nyquist rate and the upsampling and filtering 
means increases the sampling rate of the input image 
signal to at least two times the Nyquist rate, and 

(b) the means for warping includes a bilinear interpo- 
lator. 

19. An image processing system comprising: 

means for applying an input image signal to the image 
processing system, and input image signal having an 
input sampling rate; 

means for uniformly upsampling and filtering to increase 
the sampling rate of the input image signal, the 
upsampled image having interpolated values spaced 
uniformly at predetermined distances from each input 
sample; 

means for warping the upsampled image signal, and 
means for down sampling the warped image signal to 

produce a warped image output signal; 
wherein the upsampling and filtering means samples the 

input image signal at a rate of 2 N times the Nyquist rate 

where N is a positive integer. 

20. An image processing system comprising: 

means for applying an input image signal to the image 
processing system, said input image signal having an 
input sampling rate; 

means for uniformly upsampling and filtering to increase 
the sampling rate of the input image signal to at least 
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two times the Nyquist rate, the upsampled image hav- 
ing interpolated values spaced uniformly at predeter- 
mined distances from each input sample; 
means for warping the upsampled image signal, and 
means for down sampling the warped image signal to 5 

produce a warped image output signal; wherein 
the input image signal includes pixel values; and 
the upsampling and filtering means further produces four 
pixel values for each one of the pixel values in the input 10 
image signal where: 

(a) a first one of the four pixel values corresponds to 
one pixel value in the input image signal, 

(b) a second one of the four pixel values is produced by 
horizontally interpolating ones of the pixel values in 15 
the input image signal, 

(c) a third one of the four pixel values is produced by 
vertically interpolating ones of the pixel values in the 
input image signal, and 
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(d) a fourth one of the four pixel values in produced by 
horizontally and vertically interpolating ones of the 
pixel values in the input image signal. 
21. Amethod for processing an input image signal having 
an input sampling rate comprising the steps of: 

(a) increasing the sampling rate of the input image signal 
by interpolating values spaced uniformly at predeter- 
mined distances from each input sample; 

(b) warping the upsampled image signal, and 

(c) down sampling the upsampled warped image signal to 
produce a warped image output signal having an output 
sampling rate substantially equal to said input sampling 
rate, 

wherein step (a) samples the input image signal at a rate 
of 2 N times the Nyquist rate where N is a positive 
integer. 

***** 
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